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DETAILED ACTION 

1 . The Office substantially maintains the previous rejections of the claims under 35 USC 
§ 103(a) and adds an informality objection to claim 25, in light of the amendment. 



Response to Arguments 

2. Applicant's arguments filed 12/18/2008 have been fully considered but they are not 
persuasive. 

Regarding the previous rejection of the claims under 35 USC § 103(a), Applicant argues 
on pages 8-10 that the claims were improperly rejected because the Graham reference is 
available under 35 USC § 102(e) only. 

The Office respectfully disagrees. The Graham patent reference US 636981 1 was issued 
on Apr. 9, 2002, approximately 6 months before the filing dates of provisional application 
60/50641 1 and 60/506263 (i.e. Sep. 23, 2003). See the Supplemental Application Data Sheet 
filed 10/30/2007 deleting priority claims to all applications (except the provisional applications). 

Therefore, Graham is available as a 35 USC § 102(a) reference, and properly applied in 
the previous rejections of the claims under 35 USC § 103(a). 

It is further noted that any citation to specific, pages, columns, lines, or figures in the 
prior art references and any interpretation of the references should not be considered to be 
limiting in any way. A reference is relevant for all it contains and may be relied upon for all that 
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it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 
1331, 1332-1333, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 
1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). 

The Office also notes MPEP § 2144.01, that quotes In re Preda, 401 F.2d 825, 159 USPQ 
342, 344 (CCPA 1968) as stating "in considering the disclosure of a reference, it is proper to take 
into account not only specific teachings of the reference but also the inferences which one skilled 
in the art would reasonably be expected to draw therefrom." Further MPEP 2123, states that "a 
reference may be relied upon for all that it would have reasonably suggested to one having 
ordinary skill the art, including nonpreferred embodiments. Merck & Co. v. Biocraft 
Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). 

For at least these reasons, the Office asserts the rejections of the claims as set forth 

below. 

Continued Examination Under 37 CFR 1.114 

3 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1.17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.1 14, and the fee set forth in 37 CFR 1.17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on 12/18/08 has been entered. 



Application/Control Number: 10/814,844 
Art Unit: 2162 



Page 4 



Information Disclosure Statement 

4. The information disclosure statement filed 2/6/2009 fails to comply with 37 CFR 
1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent 
literature publication or that portion which caused it to be listed; and all other information or that 
portion which caused it to be listed. It has been placed in the application file, but the information 
referred to therein has not been considered. An English translation copy of the cited non-patent 
literature document CI was not included with the information disclosure statement, and therefore 
not considered. 



Claim Objections 

5. Claim 25 is objected to because of the following informalities: There appears to be a 
spelling error on line 3 of the claim, which recites "a features extraction technique". Emphasis 
added. Line 4 of the claim references "the feature extraction technique". (It is also noted that 
claim 1 recites substantially similar limitations, recites "a feature extraction technique".) 
Appropriate correction is required. See MPEP 608.01(m). 
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Claim Rejections - 35 USC § 103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth 
in section 102 of this title, if the differences between the subject matter sought to be patented and the prior 
art are such that the subject matter as a whole would have been obvious at the time the invention was made 
to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be 
negatived by the manner in which the invention was made. 

7. Claims 1-6, 8, 10, 12-22, 25, 27, 31, 33-41 and 44-45 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Scott R. Klemmer et al. ("Books With Voices: Paper 
Transcripts as a Tangible Interface to Oral Histories", CHI 2003 , Fort Lauderdale, FL, Apr. 5-10, 
2003, pp. 89-96, hereafter referred to as "Klemmer") in view of Graham et al. (US Patent No. 
6,369,81 1, filed Sep. 9, 1998 and issued Apr. 9, 2002, hereafter referred to as "Graham"). 

Regarding independent claim 1 : Klemmer teaches A computer system for generating 
a representation of time-based media, the system comprising: a feature extraction module for: 
extracting, using a feature extraction technique, features from the time-based media, the 
feature extraction technique specified by a document format specification file; (See Klemmer 
page 92 in the top paragraph of the right column discussing the creating of an MPEG-2 video 
from a video source.) and generating a media representation of the time-based media that 
represents the extracted features; (See Klemmer page 92 in the top paragraph of the right 
column discussing the making of corresponding JPEG thumbnails.) a formatting module 
communicatively coupled to the feature extraction module, the formatting module for: 
formatting the media representation according to layout parameters specified by the document 
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format specification file; (See Klemmer page 92 in the top paragraph of the right column 
discussing the creating of a paper layout from a time stamped transcript.) wherein the formatted 
media representation includes a graphical representation of a timeline and a plurality of user- 
selectable identifiers indicating locations on the timeline corresponding to the extracted 
features. (See Klemmer page 91 in the last paragraph of the right column discussing the adding 
of time code information to the print format. See also page 92 Fig. 3 showing barcodes linking 
chronological sections of a book to A/V data and page 92 in the top paragraph of the right 
column discussing the creating of a paper layout from a time stamped transcript.) 

However, Klemmer docs not explicitly teach the remaining limitations as claimed. 
Graham, though, discloses and a printer communicatively coupled to the formatting module, 
the printer for: printing the formatted media representation, (See Graham Fig. 4A #408 and 
col. 5 lines 32-50 teaching the printing of a paper reader's assistant document having an 
imprinted thumbnail and teaching additional information as being a "discussion" of user 
interests.) 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to apply the teachings of Graham for the benefit of Klemmer, because to do so allowed a 
designer to implement a mechanism to personalize a document for a particular user, as taught by 
Graham in the Abstract. These references were all applicable to the same field of endeavor, i.e., 
annotation of paper documents with electronic information. 
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Regarding claim 2: Klemmer teaches module further comprises content recognition 
software for recognizing features in the time-based media. (See Klemmer page 92 in the top 
paragraph of the right column discussing the creating of an MPEG-2 video from a video source.) 

Regarding claim 3: Klemmer does not explicitly teach the remaining limitations as 
claimed. Graham, though, discloses processing logic for controlling a printer driver interface 
associated with the printer. (See Graham col. 6 lines 4-7 discussing the use of Postscript 
printing.) 

Regarding claim 4: Klemmer does not explicitly teach the remaining limitations as 
claimed. Graham, though, discloses processing logic for controlling a printer console on the 
printer. (See Graham col. 6 lines 4-7 discussing the use of Postscript printing.) 

Regarding claim 5: Klemmer teaches wherein the feature extraction module is further 
adapted to generate the media representation in digital format. (See Klemmer page 92 in the 
top paragraph of the right column discussing the creating of an MPEG-2 video from a video 
source.) 

Regarding claim 6: Klemmer teaches wherein the feature extraction module is further 
adapted to generate the media representation in paper format. (See Klemmer page 92 Fig. 3 
and the top paragraph in the right column teaching a video paper system.) 
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Regarding claim 8: Klemmer teaches wherein at least one of the user-selectable 
identifiers comprises a barcode printed on a document displaying the media representation. 

(See Klemmer page 92 Fig. 3 showing the incorporation of barcodes on a video paper system 
document.) 

Regarding claim 10: Klemmer teaches wherein the barcode on the document can be 
scanned to play time-based media associated with the extracted features on a display device. 
(See Klemmer page 89 Abstract and page 92 in the 1 st paragraph under "Hardware" discussing 
barcode scanning and video playback.) 

Regarding claim 12: Klemmer teaches wherein the graphical representation includes 
audio content displayed as an audio waveform timeline. (See Klemmer page 92 Fig. 3 showing 
barcodes associated with Voices augmented paper transcripts in the context of the bottom 
paragraph in the right column discussing the adding of barcodes to time code metadata 
information.) 

Regarding claim 13: Klemmer teaches wherein the timeline includes markers along its 
length that correspond to user-selected segments of the time-based media. (See Klemmer page 
92 Fig. 3 showing barcode markers associated with Voices augmented paper transcripts in the 
context of the bottom paragraph in the right column discussing the adding of barcodes to time 
code metadata information.) 
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Regarding claim 14: Klemmer teaches wherein the timeline includes markers along its 
length that correspond to segments of audio content, the segments being defined by a search 
for particular features within the time-based media. (See Klemmer page 92 Fig. 3 showing 
barcodes associated with Voices augmented paper transcripts in the context of the bottom 
paragraph in the right column discussing the adding of barcode markers to time code metadata 
information.) 



Regarding claim 15: Klemmer teaches wherein the timeline includes markers along its 
length that correspond to segments of media content, at least one of the markers having text 
information describing the segment of media content. (See Klemmer page 92 Fig. 3 showing 
barcodes and associated textual passages.) 



Regarding claim 16: Klemmer teaches wherein the timeline includes markers along its 
length that each correspond to a segment of the time-based media, at least one of the markers 
having timestamp information describing the segment of the time-based media. (See Klemmer 
page 92 Fig. 3 showing barcodes associated with Voices augmented paper transcripts in the 
context of the bottom paragraph in the right column discussing the adding of barcodes to time 
code metadata information.) 
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Regarding claim 17: Klemmer teaches wherein the media representation includes a 
header describing the time-based media. (See Klemmer page 91 bottom paragraph in the right 
column discussing the placement of metadata in a header.) 

Regarding claim 18: Klemmer teaches wherein the feature extraction module is 
further adapted to generate the media representation is generated according to format 
specifications included in the document format specification file. (See Klemmer page 92 top 
paragraph in the right column discussing the creation of a paper layout based upon a transcript.) 

Regarding claim 19: Klemmer teaches wherein the format specifications included in 
the document format specification comprise a number of user-definable fields specifying a 
format of a graphical representation of the time-based media. (See Klemmer page 92 top 
paragraph in the right column discussing the creation of a paper layout based upon a transcript.) 

Regarding claim 20: Klemmer teaches wherein the format specifications included in 
the document format specification comprise a number of user-definable fields specifying a 
layout of the media representation. (See Klemmer page 92 top paragraph in the right column 
discussing the creation of a paper layout based upon a transcript.) 

Regarding claim 21: Klemmer teaches wherein the format specifications included in 
the document format specification comprise a number of user-definable fields specifying the 
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media content markers included in the media representation. (See Klemmer page 92 top 
paragraph in the right column discussing the creation of a paper layout based upon a transcript.) 

Regarding claim 22: Klemmer teaches wherein the format specifications included in 
the document format specification comprise a number of user-definable fields specifying the 
feature extraction techniques to apply to the time-based media. (See Klemmer page 92 in the 
top paragraph in the right column discussing the creation of MPEG-2 and the use of a layout 
transcript.) 

Regarding independent claim 25: Klemmer teaches A method for generating a 
representation of time-based the method comprising: extracting, using a features extraction 
technique, features from time-based media, the feature extraction technique specified by a 
document format specification file; (See Klemmer page 92 in the top paragraph of the right 
column discussing the creating of an MPEG-2 video from a video source.) generating a media 
representation of the time-base media that represents the extracted features; (See Klemmer 
page 92 in the top paragraph of the right column discussing the making of corresponding JPEG 
thumbnails.) formatting the media representation according to layout parameters specified by 
the document format specification file; (See Klemmer page 92 in the top paragraph of the right 
column discussing the creating of a paper layout from a time stamped transcript.) and wherein 
the formatted media representation includes a graphical representation of a timeline and a 
plurality of user-selectable identifiers indicating locations on the timeline corresponding to 
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the extracted features. (See Klemmer page 91 in the last paragraph of the right column 
discussing the adding of time code information to the print format. See also page 92 Fig. 3 
showing barcodes linking chronological sections of a book to A/V data and page 92 in the top 
paragraph of the right column discussing the creating of a paper layout from a time stamped 
transcript.) 

However, Klemmer does not explicitly teach the remaining limitations as claimed. 
Graham, though, discloses printing the formatted media representation, (See Graham Fig. 4A 
#408 and col. 5 lines 32-50 teaching the printing of a paper reader's assistant document having 
an imprinted thumbnail and teaching additional information as being a "discussion" of user 
interests.) 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to apply the teachings of Graham for the benefit of Klemmer, because to do so allowed a 
designer to implement a mechanism to personalize a document for a particular user, as taught by 
Graham in the Abstract. These references were all applicable to the same field of endeavor, i.e., 
annotation of paper documents with electronic information. 

Regarding claim 27: Klemmer does not explicitly teach the remaining limitations as 
claimed. Graham, though, discloses wherein extracting features from the time-based media 
further comprises performing keyword searching on the time-based media. (See Graham col. 7 
lines 50-65 discussing the use of keywords and keyphrases.) 
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Regarding claim 31: Klemmer teaches wherein the graphical representation includes 
audio content displayed as an audio waveform timeline. (See Klemmer page 92 Fig. 3 showing 
text data displayed, the particular data chosen to display having been an obvious variant. See 
also page 92 Fig. 3 showing barcodes associated with Voices augmented paper transcripts in the 
context of the bottom paragraph in the right column discussing the adding of barcodes to time 
code metadata information.) 

Regarding claim 33: Klemmer teaches wherein at least one of the user-selectable 
identifiers comprises a barcode printed on a document displaying media representation. (See 
Klemmer page 92 Fig. 3 showing barcodes printed on a video paper document.) 

Regarding claim 34: Klemmer teaches wherein the barcode on the document can be 
scanned to play time-based media associated with the extracted features on a display device. 
(See Klemmer page 92 in the 1 st paragraph in the right column under "Hardware" discussing the 
use of a barcode scanner.) 

Regarding claim 35: Klemmer teaches further comprising generating markers along 
the timeline, the markers corresponding to user-selected segments of the time-based media. 
(See Klemmer page 92 Fig. 3 showing barcodes, in the context of page 91 in the bottom 
paragraph in the right column discussing the adding of timecode metadata.) 
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Regarding claim 36: Klemmer teaches further comprising generating markers along 
the timeline, at least one of the markers corresponding to features extracted from the time- 
based media. (See Klemmer page 92 Fig. 3 showing barcodes associated with Voices 
augmented paper transcripts in the context of the bottom paragraph in the right column 
discussing the adding of barcode markers to time code metadata information.) 

Regarding claim 37: Klemmer teaches further comprising generating markers along 
the timeline, at least one of the markers including text information describing the time-based 
media. (See Klemmer page 92 Fig. 3 showing barcodes and associated textual passages.) 

Regarding claim 38: Klemmer teaches further comprising generating markers along 
the timeline, at least one of the markers including timestamp information describing the time- 
based media. (See Klemmer page 92 Fig. 3 showing barcodes associated with Voices 
augmented paper transcripts in the context of the bottom paragraph in the right column 
discussing the adding of barcodes to time code metadata information.) 

Regarding claim 39: Klemmer teaches wherein printing the formatted media 
representation further comprises printing a header describing the time-based media. (See 
Klemmer page 91 bottom paragraph in the right column discussing the placement of metadata in 
a header.) 
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Regarding claim 40: Klemmer teaches wherein printing the formatted media 
representation further comprises generating a representation in digital format. (See Klemmer 
page 92 in the top paragraph of the right column discussing the creating of an MPEG-2 video 
from a video source.) 

Regarding claim 41: Klemmer teaches wherein printing the formatted media 
representation further comprises printing a representation in paper format. (See Klemmer 
page 92 Fig. 3 showing "Books with Voices augmented paper transcripts".) 

Regarding claim 44: Klemmer teaches further comprising applying a barcode 
generation algorithm to render a barcode image including identifier information. (See 
Klemmer page 92 Fig. 3 showing barcodes that link to further information, it being implied that 
such information requires an identifier for the purposes of storing/locating that information.) 

Regarding claim 45: Klemmer teaches further comprising applying a barcode 
algorithm to render a barcode image including timestamp information. (See Klemmer page 92 
top paragraph in the right column discussing the creation of a paper layout based upon a 
transcript.) 
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8. Claims 28-29 are rejected under 35 U.S.C. 103(a) as being unpatentable over Scott R. 
Klemmer et al. ("Books With Voices: Paper Transcripts as a Tangible Interface to Oral 
Histories", CHI 2003 , Fort Lauderdale, FL, Apr. 5-10, 2003, pp. 89-96, hereafter referred to as 
"Klemmer") in view of Graham et al. (US Patent No. 6,369,81 1, filed Sep. 9, 1998 and issued 
Apr. 9, 2002, hereafter referred to as "Graham") and Ponceleon et al. (US Patent Application 
Publication No. 2003/0187642, filed Mar. 29, 2002 and published Oct. 2, 2003, hereafter 
referred to as "Ponceleon"). 

Regarding claim 28: Klemmer docs not explicitly teach the remaining limitations as 
claimed. Ponceleon, though, discloses wherein extracting features from the time-based media 
further comprises performing speech recognition on the time-based media. (See Ponceleon 
Abstract and paragraph [0003] discussing the use of automatic speech recognition for 
discovering salient sections in a speech transcription.) 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to apply the teachings of Ponceleon for the benefit of Klemmer in view of Graham, because to do 
so enabled a designer to implement a system to automatically discover salient segments in a 
speech transcript, as taught by Ponceleon in the Abstract. These references were all applicable to 
the same field of endeavor, i.e., automation information retrieval. 

Regarding claim 29: Klemmer does not explicitly teach the remaining limitations as 
claimed. Ponceleon, though, discloses wherein extracting features from the time-based media 
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further comprises performing event detection on the time-based media. (See Ponceleon 
paragraph [0007] discussing the detection of events occurring in the news.) 
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Conclusion 



9. The prior art made of record and not relied upon is considered pertinent to applicant's 
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